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ABSTRACT 

This paper reports on several generic evaluation 
methods used by the evaluation office of the Curriculum Research and 
Development Group of the University of Hawaii (EO-CROG). Scrivens's 
(generic) checklist approach was taken one step further by developing 
generic data collection instruments to be used with any evaluation 
method: (1) a formative evaluation feedback questionnaire; (2) an 
observation coding sheet; and (3) a teacher instructional behavior 
record. The student questionnaire is based on an evaluation form from 
an American Educational Research Association workshoff Scriven's 
checklist and BO-CRDG experiences. The one page forib includes 
Likert-type items on course (or class) organization, pace, 
difficulty, and opportunities to interact or ask questions. Three 
open-ended questions cover strengths, weaknesses, and other comments. 
Students assign a letter grade (A-F) to the course. The observation 
foriL used is a modified version of the Far West Laboratory for 
Educational Research and Development. It includes engagement time, 
active teaching behaviors, student and teacher field notes, and 
teacher time allocation. Using the State's performance expectations 
for major subject matter areas, cognitive tests were constructed. For 
greater efficiency, final reports would include writings that had 
proven to be successful in communicating in the past. The three 
generic instruments developed in this study are appended. (PN) 
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Generic Evaluation--An Approach to Surviving Multi-Project Evaluations 

introduction: 

At last count our evaluation office was responsible for 13 
separately furded evaluation rirooects. Granted the range of funding 
was modest: S^,000 to $31,000; however, as most evaluators know, all 
evaluations require substantial amounts of affort (often more than 
originally expected) regardless of funding level. As the senior staff 
of the evaluation section of our curriculum research and development 
organization for the past 10 years, we have learned to adjust to the 
practicalities of conducting evaluations of several projects at the 
sa»e time* Only recently have we had the time to analytically study 
our efforts in this area. 

In this paper we describe several areas where adjustments were 
critical. The situation is akin to a meta-^evaluation of an evaluation 
organization's procedures. Because of the relative newness of the 
topic, we will focus on what would logically be called formative 
meta-evaluation of organizational procedures. 

Although many evaluators spend substantial amounts of time . ' 
developing data collection instruments "from scratch,'* others have 
argued that as much as possible, evaluators and resMrchmrB should try 
to use existing instruments. It is entirely reasonable to consider 
the possibility tha^ a newly developed instrument may not have as much 
quality as an older one. 

At a somewhat gros."* level, there is generalized use of developed 
methodologies; for example, -che use of Likert*-type scales in which 
dagrees of agreement are assigned numerical values. Likewise! the use 
of a standard set of adjectives has been used with the semantic 
differential approach. As a final ex*ample we cite the plethora of 
evaluation modelt^ which can be used to evaluate disparate programs or' 
products . 

When Scriven first presented his checklist approach (subsequently 
modified to the **key evaluation checklist"), he asserted that the 
method could be applied to any of the existing evaluation models or 
approaches. He argued, for example, that all evaluations should be 
concerned about the need for the program or product. Likewise all 
evaluations should look at critical competitors even if such 
competitors have to be invented. We took Scriven^ s (generic) 
checklist approach one (critical) step f ^-ther by developing generic 
data collection instruments that could be used with any evaluation 
method, including of course the checklist approach itself* 
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DATA-COLLECTION INSTRUMENTS 



Queationnairea: Much muat precede the adniniatration of 
queationnairea: deaign of questionnaire, typing of draft, pilot 
teating, item analyaaa, aolicitation cf feedback, reviaion, retyping, 
and reproduction. Although word proceaaora have made it poaaible to 
make reviaiona auch more quickly, the creation of unique 
queationnairea ia atill a tissie-conauming taak. Several years ago we 
caae acroaa a generic evaluation form that waa uaed to obtain 
feedback on American Educational Re<aearch Aaaociation wo.'^kshopa. The 
form had undergone aeveral evolutiona and waa uaed for aomewhat 
diaparate workahopa. 

Becauae the great majority of our pr electa were deliv€«red 
through claasrooma, there were generic terma auch aa **courae" or 
"claaa** which would apply to virtually all of our projecta. The 
field of evaluation haa an example of a generic approach in the Key 
Evaluation Checkliat, which according to Scriven will work with any 
product or pro3ect aa well aa any eve.iuation approach or model. He 
aaw the checkliat aa being uaeful in enauring that evaluatlona 
include or ^t leaat c:onaider the ma^or important aapecta 
character i at ic of good evaluationa. In the midat of poaaible 
impending chaoa in our ahop, we aaw ' he generic approach aa alao 
providing a meana of eliminating aubatantial efforta in deaigning and 
producing queationnairea* 

We then combined the AERA workahop evaluation form, Scriven'a 
checkliat, and our vaat experience in conducting evaluationa and came 
up with a atudent queationnaire that we have been able to uae with 
pro3ecta involving auch disparate topica aa bilingual education, 
computer education, apecial education, and nutrition education. The 
form ia one page and includea Likert-type itema on course (or claaa) 
organization, pace, difficulty, opportunitiea to interact or aak 
queationa* There are three open-ended queationa covering strengtha, 
weakneaaea, and other commenta. Finally the atudent is aaked to 
aaaign a letter grade (A-F> to the courae. A copy of the instrument 
ia in the appendix* 

Obaervations: Recently observation techniquea have been 
developed that enable the collection of ayatematic, reliable data 
from the classroom. During the early part of thia decade, the US 
Offi'^e of Education awarded a multi -million dollar contract to the 
Far Weat Laboratory for Educational Research and Development to 
conduct a descriptive study of bilingual education. The naturaliatic 
approach uaed by the Far Weat Lab included an extenaive battery of 
obaervational instruments baaed on the exiating literature. Although 
the training of the data collectora for that atudy waa extensive and 
expenaive (80 houra each for aix data collectora), it became clear 
that a nuch more modeat level of training atill could yield valid, 
reliable data. 

The following (modified) subset of the observational forma used 
in the Far Weat Lab atudy waa aelected: a) engagement time (cf. time 
on task or academic learning time), b) active teaching behaviora, c) 
atudent field notea, d) teacher field notea, and aometimea e) teacher 
time allocation. All £ormm were found to be workable as well aa 



relevant to evaluationa of projects in differing areas. Instruments 
covering engagement tine and active teaching behaviors are presented 
in the appendix. 

Cognitive Testing: It is much more difficult to escape having 
to customize cognitive test development; however, even here there are 
some efficiency promoting methods. We have developed, for example, a 
computer literacy test that has been used in computers-assisted 
education projects as well as computer literacy projects. 

By using the Staters performance expectations for the major 
subject matter areas, we were able to construct cognitive tests that 
attempted to measure what were the ** official" objectives of the 
Department of Education. No matter what a given project professed as 
objectives, valuable evaluation information could be obtained by 
investigating the project ^s effect on the Department's professed 
objectives. 

Report Writing: No matter how carefully we planned, it almost 
always seemed that our final reports were late. Then we realized 
that a generic approach to report writing would also make aens«(. For 
one thing there would be less energy spent on rewriting boilerplate 
sections such as strengths and weaknesses of the evaluation staff or 
on observation methodology* By the evolutionary process^ writings 
that had proven to be successful in communicating in the past would 
be inclu<led. Finally typographical errors would be greatly reduced 
inasmuch much of the report will have gone through previous 
proofreading • 

SUMMARY /CONCLUSION : 

When an evaluation organization is responsible for several 
projects^ it may be desirable to use generic evaluation methods that 
can cross over disparate content areas. Which generic methods are 
effective can best be ascertained by formative and r^ummative 
evaluation procedures. In this paper we report on several generic 
evaluation methods that have proven effective (or at least have 
enabled us to survive) and suggest that the field ought to seriously 
consider the development and dissemination of others that would 
assist those evaluation organizations that have to conduct several 
evaluations at the same time. 

We see^an additional benefit to using generic evaluation forms. 
It may be possible to conduct genuine national studies, where sites 
use comparable forms. Furthermore, the use of generic forms even 
for organizations concerned with only one evaluation project will 
minimize the occurrence of flaws in the wording of the items. 

The attached generic forms have proved to be useful for our 
situation. Perhaps in order to be effectively used by others they 
would have to be customized to better fit the situation as well as 
the philosophy of the (other) evaluation Btm££ » On the other hand 
generic evaluation methods may be the key to promoting efficiency and 
quality in evaluation. How will we know what the true situation is? 
By evaluating, perhaps even generically, the use of generic 
evaluation methods. 



Fornative evaluation feedback from arudents 
SEX: Male Female AGE 



Directions: Pleaae answer the following questions in relation to your 
recent experiences in the program. 



1. The organization of the course has been: 

excellent 12 3 4 5 

2. Overall the pace of the course has been: 

too slow 1 2 3 4 5 

3. Opportunities for asking questions have been: 

insufficient 12 3 4 5 

4. The amount of work required of you h^s been: 

far too much 12 3 4 5 



poor 



too fast 



sufficient 



far too little 



5. 



6. 



7. 



Did you understand what was going on in claBB7 
almost never 12 3 4 5 



almost always 



Considering what you have gotten from taking this course, the 
time spent was: 



well worth it 1 2 3 4 

What were the strengths of the course? 



not worth it 



Qm Whet were the weaknesses of the course? 



9« Give the course an overall grade (A~F> « 
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TEACHER INSTRUCTIONAL BEHAVIOR RECORD 

SITE: 1 2 3 4 5 6 7 8 9 CLASS NUMBER: 

rOOAV'S DATE: OBSERVER NUMBER: 

LESSON TYPE: 12 3 4 5 APPROXIMATE LESSON DURATION: 

Below are listed those teacher behaviors which may have occurred during 
the time you observed Instruction for this lesson. For each, place a 
checK mark In the appropriate box If It occurred (even If It occurred 
only once, you should check the boxf* 

1« The teacher actively presented Instructlon/infonnatlon 

I I Stated what students were to learn In lesson (goals, objectives) 

T~T Outlined the lesson before proceeding 

I I Explained : concepts, definitions, relationship of tasks to goals, etc. 

I I Reviewed : goals, previous related instruction, etc. 

I I Illustrated : how to do the work, how to do a problem, etc. 

"FT Questioned students to see if they understood 

I I Answered student's questions about what they were to do 

I I Summarized : what v/as presented, what class had done or learned, etc. 

I I Moved the class quickly from one activity or lesson to another 

2. The teacher established and maintained engagement of students 
in instruction, tasks, activities 

I I Told students to attend to tasks (whole class or individually) 

I I Explained the rules of behavior 

I I Signalled students to get to work (turned off llgnts, eye contact, etc.) 

I I Resolved potential disruptions 

I I Resolved student misbehavior 

I I Encouraged students to keep up (maintain pace) 

I I Sustained momentum in the lesson, not letting It slow down 

Tnr Adjusted instruction (faster/slower) according to students' speed 
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TEACHER INSTRUCTIONAL BEHAVIOR RECORD , p. 2 

3. The teacher monitored students' progress in learning, completing tasks 
I I Scanned the room to see if everyone was working 

I I Reviewed students' work when it was completed 
I I Recorded students' work when it was completed 
{ I Monitored students' responses 
J^X ^^o^ed tf'e foom, checking students' work 

1 T Questioned students ; learned a concept, learned a fact, completed work 
T T Collected students' work 

4. The teacher provided instructional feedback to students 
I I Told student answer (work) was correct or incorrect 

I I Provided "key" so students could check answers 
XZL Model ed appropriate responses for students 
I I Demonstrated how to compl ete work correctly 

5. The teacher focused on developing students' language (LI or L2) 
I I Insisted on whole sentence utterances 

I I Asked comprehension questions both in LI and L2 if necessary 

I I Rejected students* response in one language when focus was other language 

I I Demonstrated differences of concepts between LI and L2 

I I Corrected students' use of either LI or L2 

6. The teacher used two languages for instruction 
1 T The teacher used formal translation 

I I The teacher used informal translation 
I I The teacher used language alternation 

I r 7. The teacher responied to ci'ltural cues and used these to further 
instruction 
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